Skip to content

Add keypoint GUI for manual calibration#1749

Open
romleiaj wants to merge 34 commits into
dev/sealtkfrom
sealtk/keypoint-gui
Open

Add keypoint GUI for manual calibration#1749
romleiaj wants to merge 34 commits into
dev/sealtkfrom
sealtk/keypoint-gui

Conversation

@romleiaj

@romleiaj romleiaj commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

No description provided.

@romleiaj romleiaj marked this pull request as draft July 9, 2026 21:17
@romleiaj romleiaj mentioned this pull request Jul 9, 2026
@romleiaj romleiaj changed the base branch from sealtk/calibration to sealtk/image-registration July 10, 2026 16:53
@romleiaj romleiaj force-pushed the sealtk/image-registration branch from 30ebbee to a6d545d Compare July 10, 2026 18:38
@romleiaj romleiaj force-pushed the sealtk/keypoint-gui branch 2 times, most recently from f25924b to 0de24f9 Compare July 12, 2026 15:12
romleiaj and others added 24 commits July 12, 2026 11:49
homography.ts: 3x3 matrix primitives (multiply, invert, apply, linear
and DLT solvers) plus warp-grid subdivision helpers. transform.ts:
keypointgui-style transform models (translation / rigid / similarity /
affine / homography) that all estimate to a plain Matrix3, with
Similarity as the single shared default. CameraCalibrationStore: the
persistence core for camera-rig calibration -- per-pair homographies,
correspondences, transform types, and producer provenance, with
hydrate() from dataset meta, dirty tracking, and the portable
calibration.json round trip (toCalibrationJson / loadCalibrationText).

Interactive creation (point picking, fitting, overlay preview) is not
part of this branch; it layers on top in the Manual Alignment GUI.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Persist cameraHomographies / cameraCorrespondences /
cameraTransformTypes / cameraCalibrationSource in dataset meta on both
platforms (server allowlist + desktop meta merge), and load
calibrations produced externally (e.g. by kamera) at import time: the
multicam import dialog gains a per-camera transform-file picker, the
desktop backend parses DIVE calibration .json pairs
(fromCalibrationPairs / toCalibrationPairs) and auto-discovers a
calibration.json in the import parent folder, and saveMetadata writes
the portable calibration.json alongside the project so the calibration
travels with the dataset.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
When every camera resolves a transform into the reference camera's
space (first camera in display order, composed through the calibration
pair graph), an Align button warps each pane's imagery and annotations
into that shared space at draw time -- stored geometry always stays
native per camera. Includes: AlignedViewStore + resolution wiring in
the Viewer; AlignedImageLayer (sub-quad warped imagery with seam
overlap); display-transform routing through every annotation layer;
transform-aware linked pan/zoom (useAlignedNavigation/useLinkedViewers,
with the raw screen-delta camera sync retired once transforms exist);
drawing/editing on any camera with edits mapped back through the
inverse; and a continuous cross-camera mirror that re-projects every
geometry edit onto all other calibrated cameras.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Importing single-camera annotations into a calibrated multicam dataset
now offers an "Import to all cameras" checkbox: after the import lands
on the active camera, every track is copied onto each other camera
through the camera-to-camera homographies (bounds, rotated bounds, and
GeoJSON geometry), then saved. Unlike the Align View continuous mirror
this is a data import, so confidence pairs and track/feature attributes
are copied too. Gated on the whole rig resolving to the reference
(alignedView.available), independent of the display toggle.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The dataset's saved calibration moves from a single all-pairs
calibration.json to one calibration_<camera>.json per non-reference
camera (a pair not touching the reference files under its right
camera), matching the per-camera files users already handle from
kamera. File names are discovery/provenance only: the pair's left/right
names inside each file body stay authoritative, so a misnamed or copied
file can never rebind a transform to the wrong camera. The legacy
single file is still read and is migrated (rewritten per-camera, then
removed) on the next save; stale per-camera files are cleaned up so the
on-disk set always mirrors the saved calibration.

Per-file producer stamps are compared when the set is merged (project
load and import seeding): agreement keeps the stamp, disagreement
becomes a { mixed: true, files: {...} } composite that the Align View
tooltip surfaces as a mixed-generation warning instead of composing
silently. A save never writes the composite into per-camera files --
that would read back as a unanimous rig.

Parent-folder transform discovery now returns every self-identified
calibration file (calibration.json first, then per-camera files), and
the multicam import dialog auto-attaches calibration_<camera>.json to
its matching camera slot. Files written by DIVE now also self-identify
with type: dive-camera-calibration so discovery recognizes them.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The per-camera calibration_<camera>.json files are now the only on-disk
form: the single all-pairs calibration.json is no longer read, migrated,
or given priority during import discovery. The Export menu gains explicit
calibration export -- one file per calibrated camera, or all of them
zipped -- built from the same per-camera grouping the save path uses, and
the multicam Everything zip now carries the files too. Exports resolve
the calibration the same way loading does (standalone files first, the
import-time meta seed otherwise) and never stamp a file with a mixed
composite source.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Calibration export is now strictly per-camera -- one
calibration_<camera>.json at a time, no all-cameras zip -- and works on
both platforms: desktop writes through the backend as before, web builds
the file client-side from the dataset meta and downloads it. The Import
menu gains a matching per-dataset calibration import (same workflow as
annotation import): pairs the file names replace the current ones, other
cameras' pairs are kept, so per-camera files can be imported one at a
time to assemble a rig; the viewer store rehydrates immediately so the
Align View picks up the new transforms without a reload. Grouping and
merge semantics live in a new shared vue-media-annotator module used by
the desktop backend, both Export menus, and both import paths, so listed
cameras, written files, and merge behavior cannot drift between
platforms. Web needs no server changes: the calibration meta fields were
already allowlisted.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Import and Export buttons now advertise "Supplementary Data" in
their hover tooltips instead of "Annotation Data", since both menus
carry calibrations as well. The Import Calibration blurb shrinks to one
line. The warp-on-import checkbox moves out of the active-camera alert
to sit beside Overwrite as "Warp to All", always visible on multicam
datasets and greyed out until the whole rig resolves, with an
"N/M cameras calibrated" hint. The viewer publishes that count into
AlignedViewStore from the same computation as the Align View tooltip,
so the two indicators can never disagree.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The generic Import Calibration button becomes one button per
non-reference camera pair ("Import eo -> ir"), each scoped to its
camera: only the file's pairs naming that camera are taken, they
replace that camera's current pairs, and other cameras' pairs are
kept -- so per-camera files assemble a rig one at a time and a
multi-pair file pointed at one camera doesn't drag the rest along.
Scoping to a camera the file doesn't name refuses with a clear error.
Buttons are green when the camera already has a calibration and yellow
when it doesn't, and importing onto an existing one asks for
confirmation before replacing it, so no separate overwrite toggle is
needed. The camera filtering and pair/camera summary live in the
shared calibration-file module so desktop and web behave identically.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ormat

"Calibration" collided with the pre-existing stereo camera file (an
intrinsic/extrinsic camera model for the measurement pipeline), and in
computer vision this process is image registration, so the feature says
registration everywhere: "Import Registration" / "Registration: <camera>"
menu sections, importCameraRegistration and exportCameraRegistration API
members and IPC channels, registrationProgress ("N/M cameras registered")
published on the aligned-view store, matching error messages, the
internals (CameraRegistrationStore, cameraRegistrationFiles, and the
load/merge/build helpers), and the wire format: the dataset meta field is
cameraRegistrationSource, renamed through the server models, allowlist,
and tests alongside the RegistrationSource type and stamp helpers. Files
are named <camera>_to_<reference>_registration.json and self-identify as
dive-camera-registration: each file states the destination of the mapping
it carries (ir_to_eo_registration.json registers ir onto eo), making the
reference-camera rule -- first camera in display order -- legible from
the artifacts themselves. The name scheme contains neither "calibration"
nor "cal", so these files can no longer be mis-suggested as stereo camera
files by the stereo parent-folder discovery heuristic. The Align View
keeps its name (registration is the data and process; the Align View is
the display mode that renders registered imagery), and the stereo
camera-file surfaces are untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The registration reference was implicitly the first camera in display
order, which merely coincided with the camera users pick at import in the
common EO-first case. The import dialog's picker is now labeled
"Reference Camera" ("Choose which camera to use as the reference camera
for image registration") and its choice -- stored as the dataset's
defaultDisplay -- actually drives the reference, through a shared
referenceCameraName() resolver used by the viewer's transform resolution
and N/M-registered status, both Export menus, and the desktop per-camera
file persistence, falling back to display order when the choice is
missing. The viewer now publishes the reference on the aligned-view store
even before the rig resolves, so the per-pair import buttons name it
("Import ir → eo") instead of assuming camera-map order.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A file named *_registration.json with a pairs list now qualifies without
the self-identifying type marker -- the conventional name is the
producer's declaration of intent, matching every other load path where
type is optional. Non-conventionally-named JSONs still require the exact
type string so stray JSON in the collect root is never grabbed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pair bodies are authoritative on load, so a pair naming a camera the
dataset doesn't have imports fine but never resolves in the Aligned
View. beginMultiCamImport now collects a warning per transform file
whose pairs name cameras outside the dataset, carried on the import
response as importWarnings and shown in the import confirmation dialog
so the mismatch is visible before Finish Import.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Web parity for the desktop-only transform seeding: the import dialog's
per-camera transform pickers and parent-folder auto-discovery now work
on web (Files stashed in the multicam registry, discovery scanning the
selected folder's root with the same qualification rules as the desktop
backend), and the upload flow parses the attached registration files up
front, seeds the new dataset's saved camera registration through the
meta PATCH, and prompts the same unknown-camera warning as desktop.
mergeRegistrationSources moves to the shared cameraRegistrationFiles so
both platforms compose producer stamps identically.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Each collect's root is scanned for DIVE registration .json files (the
same qualification rules as the single import's parent-folder
discovery) on both platforms, and the shared batch scan assigns them to
camera slots on the collect's importArgs via the slot-matching rule now
extracted from the import dialog. The batch table shows what was
attached, warns when a file has no free slot, and surfaces the
unknown-camera seeding warnings per collect row -- batch imports skip
the confirmation dialog that normally shows them.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The warp renders a snapshot of the camera's displayed image element and
only re-rendered on transform or frame changes, so a same-frame
presentation change -- the percentile-stretch URL remap swapping the
quad's <img>, or the first enhancement toggling the CSS filter -- left
the warped pane stale until the Aligned View was exited. The media
controller now exposes an imageRevision counter that the annotators
bump on those redraws, and LayerManager re-renders the warp when it
moves.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
romleiaj and others added 9 commits July 12, 2026 11:52
The web and desktop discovery paths each carried their own copy of the
registration-file qualification rule, candidate ordering, and file-name
pattern; share them from dive-common/registrationParentFolder. The
unknown-camera import warning was likewise duplicated between the desktop
and web seeding paths; share it as unknownCameraWarning next to
mergeRegistrationSources, with desktop deriving named cameras from the
pairs' own left/right fields instead of re-splitting transformTypes keys.
Drop the mergeRegistrationSources re-export shim in favor of importing it
from its home in cameraRegistrationFiles.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
AlignedImageLayer polled with a bounded requestAnimationFrame loop to
catch the annotator swapping its <img> element asynchronously, because
no event reached the layer. imageRevision is now that event: drawImage
bumps it on every media-quad redraw (including the post-seek swap), and
the LayerManager watch re-renders the warp from it. VideoAnnotator now
bumps it on the initial video quad draw, the one element swap a video
pane ever makes.

This also removes the poll's failure mode: it gave up after ~60 frames,
so an image that loaded slower than a second left a stale warp until
the next trigger; the revision bump fires whenever the image actually
lands.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The LayerManager watch that re-renders the Aligned View warp on
imageRevision bumps called alignedImageLayer.update(), but
alignedImageLayer was never destructured from useLayerManagerAlignedView
-- it sat inside the ...alignedViewHelpers rest object, so the callback
threw on every bump and Vue swallowed the error. The warp therefore
never followed image-enhancement changes (percentile-stretch toggles
left it stale until the Aligned View was re-toggled); the removed rAF
poll had been masking the dead watch for post-seek swaps.

Move the watch into setupDisplayTransformWatches, where the layer is
genuinely in scope and the transform/frame/revision triggers live
together, and pin the contract with a composable-level spec (bump
imageRevision with a swapped element -> the warp re-renders from it).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The center-view button (and the r hotkey) reset every pane to its own
native image bounds, but while the Align View renders panes in the shared
reference space that parks the view on an empty corner -- and each pane's
reset events ran through the identity link, so whichever pane landed last
dragged every other pane along with it.

The aggregate reset now consults an override installed by
useAlignedNavigation: while the aligned view is active, fit the SELECTED
camera's content (its native bounds mapped through its native->reference
homography) in its own pane, and mirror that view to every pane through
the existing link. Falls back to the native per-pane reset when the
aligned view is off.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Extend CameraCalibrationStore with the Manual Alignment creation state
and operations: active pair, point picking (add/pick/move/drag-refine,
select/delete, clear last/pair), transform fitting (fitTransform /
maybeFitPair via the transform estimators), the ghost-overlay alignment
mode/opacity, linked-nav toggle, and cursor/recenter plumbing.
CalibrationKeypointLayer renders the picked correspondences (locked
blue / unlocked yellow, exact-pixel markers, click-select and drag) and
the cross-pane ghost warp preview; CameraImage moves here with it.
useCalibrationNavigation links the active pair's panes through the
fitted homography while picking (standing down under the Align View).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CalibrationTools.vue (modeled on VIAME's keypointgui): pick point pairs
across a camera pair, fit the chosen transform model with a
fit-readiness countdown, preview the fit as a ghost overlay with warp
opacity, review rig alignment status (N/M cameras calibrated), and
save/load the calibration -- to dataset meta and the portable
calibration.json. Wired into the Viewer as a context panel: workspace
chrome minimizes while calibrating, only the active pair's panes show
on 3+ camera rigs, the general Align warp suspends while picking so
picks are always native-space, and LayerManager mounts the keypoint
marker/ghost layer per pane.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rename the feature's own files and identifiers to the registration
vocabulary the rest of the branch adopted: CalibrationTools ->
RegistrationTools (CameraRegistration/), CalibrationKeypointLayer ->
RegistrationKeypointLayer, useCalibrationNavigation ->
useRegistrationNavigation, and the calibration-named locals, props, and
UI strings that referenced them. Stereo calibration (a genuinely
different artifact) keeps its name.

Reuse shared helpers where the feature had its own copies: the panel's
unknown-camera notice now comes from unknownCameraWarning, and the
geojs warp-quad mapping duplicated between AlignedImageLayer and the
keypoint ghost is extracted to geojsWarpQuads in alignedView/homography.
The linkedViewers README's "planned" registration-pair-link section now
describes the shipped composable.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same replacement as the Aligned View warp: the ghost polled with a
bounded requestAnimationFrame loop to catch the source pane swapping
its <img> element after a seek. The source annotator now bumps
imageRevision on that swap, so LayerManager watches the ghost source
camera's revision (alongside its frame number) and re-renders the
ghost from the element actually on screen -- with no ~1s give-up
window on slow-loading frames.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rename the Manual Alignment panel to Camera Registration and scope it to
its actual job -- picking points, fitting, and saving:

- Drop the panel's "Load registration" button: the Import menu's
  per-camera registration targets are the one import path.
- Drop the panel's export button too: the Export menu's per-camera
  registration downloads (web and desktop) are the one export path. The
  refined-from-source notice now points there (save first -- the menu
  reads saved meta).
- Remove CameraRegistrationStore.toRegistrationJson, the last whole-rig
  (all pairs in one file) serializer; per-camera
  <camera>_to_<reference>_registration.json files are the only
  registration file format. loadRegistrationText stays as the shared
  parser, and its round-trip specs now serialize through the production
  buildPerCameraRegistrationFiles.
- Protect unsaved panel edits on exit: the browser beforeunload warning,
  the navigate-away prompt, and the desktop close dialog now fire on
  registration dirt too (hasUnsavedChanges = pending annotation saves OR
  a dirty registration), and the desktop dialog's Save choice persists
  the registration via the new Viewer.saveRegistration.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@romleiaj romleiaj force-pushed the sealtk/image-registration branch from b60430b to 1d9a2eb Compare July 12, 2026 15:54
@romleiaj romleiaj force-pushed the sealtk/keypoint-gui branch from eca04d4 to 55f0eaa Compare July 12, 2026 15:55
@romleiaj romleiaj marked this pull request as ready for review July 12, 2026 15:56
@romleiaj romleiaj requested a review from BryonLewis July 12, 2026 15:56
A registration imported from a producer (e.g. KAMERA) carries matrices but
no picked points, and the panel treated it as incomplete: "Fit pan/zoom"
stayed greyed out (it gated on point count, though the link only needs a
homography), the status line claimed points were still needed, and picking
was force-enabled so stray clicks placed points on a registration that
needed none.

- Add a three-state transform status (loaded from file / fit from N point
  pairs / none) and gate the pan/zoom link on having a transform at all,
  renaming it "Link pan/zoom"; enabling it fits the pair on demand.
- Make picking an explicit "Pick points" toggle scoping the authoring UI
  (cursor readout, correspondences, transform type), defaulting per pair:
  off in review of a file-loaded transform, on when points are needed.
- Decouple linked navigation from picking mode -- it now runs whenever the
  panel has a pair and the link is on; the panel clears the pair (and with
  it the warp ghost) on close to keep that scoped.
- After a registration import, summarize what the loaded transform drives
  and, when the panel is open, re-select the imported pair in review
  posture (hydrate cleared it and left the panel inert).
- Confirm before "Save registration" overwrites existing per-camera
  registration files, naming exactly the files whose content changes --
  untouched pairs rewrite byte-identical and don't warn -- along with the
  provenance stamp being replaced.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Base automatically changed from sealtk/image-registration to dev/sealtk July 13, 2026 20:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants